The Hidden Challenge Behind AI Personalization

Personalization has become one of the defining features of the digital economy. When we open a music app, browse an online store, search for hotels, or scroll through a content platform, an algorithm is deciding what we see next. These decisions shape what we buy, what we watch, where we travel, and how long we stay engaged.

It is tempting to think of personalization as a prediction problem: learn what a user likes, then recommend similar items. For many years, this view has been remarkably successful. Platforms learned to represent users and products as points in a shared space. If a user was close to a product, song, hotel, or article in that space, the system treated it as a promising recommendation. This approach was fast and intuitive. Even when a platform had millions of items, it could search efficiently for items most similar to a user’s past behavior.

But modern personalization is becoming more ambitious. Users do not experience recommendations one item at a time, but as sets and sequences: a playlist of songs, a page of hotels, a feed of videos, a collection of products, or a series of news stories. In these settings, the value of one recommendation often depends on what appears around it.

Why Is “What You Liked Before” No Longer Enough?

A song may be enjoyable on its own, but a series of similar songs can become repetitive and uninteresting. A hotel may look attractive in one comparison set but less compelling next to a cheaper alternative. A product may be more appealing when paired with a complementary item, or less useful when shown beside a close substitute. In other words, context matters.

This is one reason transformer-based AI models are so appealing for personalization. Transformers are well known for their success in language models, where the meaning of a word depends on the words around it. A similar idea applies to user behavior. The next item a user wants may depend not only on their general preferences, but also on the sequence of items they have already viewed, clicked, purchased, skipped, or saved.

For businesses, this opens an exciting opportunity. More expressive AI models can capture richer patterns in customer behavior. They can account for variety, repetition, complementarity, substitution, and sequence effects. They can move personalization beyond “show me items similar to what I liked before” toward “show me the right combination of items at the right moment.”

The hidden challenge, however, is that better prediction does not automatically lead to better decisions. A platform still has to decide what to recommend, and it has to do so almost instantly. A music app cannot evaluate every possible continuation of a playlist, and likewise, a travel platform cannot test every possible group of hotels before loading a search results page. This is the operational side of AI personalization. The model may understand users better, but the business still needs an algorithm that can quickly turn that understanding into a recommendation that is useful. If the model is too complex to optimize, its predictive power may be difficult to translate into real value.

Simplicity by Design

My research studies this gap between prediction and decision-making. In particular, I focus on a class of models called simple transformers. These models use a single self-attention layer, making them much simpler than the large, multi-layer transformer architectures often associated with modern AI. The goal is not to build the largest possible model. The goal is to identify a model that is expressive enough to capture important customer behavior, but structured enough to support fast decision-making.

This tradeoff is central to business applications of AI. In many commercial settings, the best model is not necessarily the most complex one. The best model is often the one that balances predictive accuracy, computational speed, and decision quality.

Simple transformers offer one way to strike this balance. They can capture interactions among recommended items, such as whether two products complement or substitute for one another, or whether a sequence provides enough variety. These are effects that traditional embedding-based methods often miss. At the same time, simple transformers have enough structure to allow more principled optimization.

The algorithmic idea follows a two-stage process common in real-world recommendation systems. First, the system retrieves a smaller set of promising candidates from a very large catalog. Then it ranks those candidates more carefully, taking into account the contextual effects captured by the transformer.

How Do We Optimize Millions of Choices in Real-Time?

The challenge is that both stages need to be designed for the transformer model, rather than simply borrowing methods developed for older recommendation systems. If retrieval ignores context, it may discard items that would be valuable in combination. If ranking relies only on greedy search, it may miss better sets or sequences. My research develops methods that address both stages: retrieving candidates in a way that preserves the value of the best recommendations, and ranking them using the structure of the simple transformer.

The theoretical results show that, under appropriate conditions, this approach can achieve near-optimal recommendations in sublinear time. In practical terms, the algorithm does not need to examine the entire catalog. This matters because modern platforms often operate at enormous scale, where even linear-time search can be too slow for real-time interaction.

We also tested the approach on large datasets from Spotify, the audio streaming service, and Trivago, a service that aggregates and compares prices for hotels and accommodation. The results suggest that simple transformers can substantially improve prediction accuracy compared with non-attention models, while remaining close to more complex transformer architectures. In recommendation experiments, the proposed algorithm also improved performance over standard retrieval and ranking methods, showing that both stages contribute to better decisions.

“The real business value comes when predictions are converted into decisions: which products to show, which hotels to rank higher, which songs to recommend next, which content to place in front of a user.”

Turn Predictions Into Decisions

The broader lesson is that AI in business should not be evaluated only by how well it predicts. Prediction is only the first step. The real business value comes when predictions are converted into decisions: which products to show, which hotels to rank higher, which songs to recommend next, which content to place in front of a user.

As AI systems become more powerful, this connection between learning and decision-making will become increasingly important. Companies will need models that understand complex patterns in customer behavior, but they will also need algorithms that can act on those patterns quickly, reliably, and at scale.

Personalization is one example, but the same issue appears across many business domains: pricing, inventory, supply chains, advertising, and resource allocation. In each case, AI can help predict what may happen next. The harder question is what to do next. That is the hidden challenge behind AI personalization. The future will belong to systems that make better predictions and also turn those predictions into better decisions.

Further Reading

Lin An, Andrew A. Li, Vaisnavi Nemala, Gabriel Visotsky, “Near-Optimal Personalization with Simple Transformers,” https://arxiv.org/abs/2503.00608